Method and apparatus for image data transfer in digital photographing

ABSTRACT

An apparatus and method are described in which image information is received from an image sensor relating to one image frame, one or more sub-images are produced from the received image information according to given cropping data; and the produced one or more sub-images are forwarded through an output.

TECHNICAL FIELD

The present application generally relates to image data transfer indigital photographing.

BACKGROUND

In digital photography, images are typically formed by passing lightfrom an object through an objective onto an image sensor such as a CMOSor CCD unit. The image sensor typically comprises millions of pixeldetectors for forming corresponding pixels of digital images.

In typical digital cameras, pixels or picture elements are formed by acamera module using pixel detectors sensitive to three different maincolors (typically Red, Green and Blue). Images are then formed using thegained pixels. In the forming of the images, image quality is enhancedby various computational operations such as edge enhancement and pixelcolor interpolation. Image files are formed typically with imagecompression. Also some adjustments are typically made in white balance,focus and exposure by controlling the operation of the camera module.The processing is often distributed such that the camera module producesonly elementary data that is passed over a fast camera module interfaceto another processor for heavier calculation.

SUMMARY

Various aspects of examples of the invention are set out in the claims.

The inventor realized that the development of image sensors is leadingto new difficulties in the data transfer of image information from acamera unit to subsequent use in a camera device. In battery-operatedmobile devices, power consumption is proportional to the clockfrequencies used. The higher speeds, the higher power consumption. Ifimage sensors develop radically over those presently used in cameraphones e.g. to 40 megapixels and more, the data bus between a cameraunit and its host device should be very fast and/or the image datashould be compressed already in the camera unit.

According to a first example aspect of the present invention, there isprovided an apparatus, comprising:

an input configured to receive image information from an image sensorrelating to one image frame; and

an image processing circuitry configured to produce one or moresub-images from the received image information according to givencropping data and to forward the produced one or more sub-images throughan output.

The image sensor and the apparatus may be parts of a camera unit. Theoutput may be a camera unit interface configured to exchange informationbetween the camera unit and a circuitry external to the camera unit.

The image processing circuitry may be further configured to send imageinformation only with the produced one or more sub-images.

The cropping data may further comprise exposure definition information.The image processing circuitry may be further configured to control theimage sensor to produce image information with exposure timecorresponding to the exposure definition.

The image processing circuitry may be further configured to defineindividual lines of pixels for image capture for the image sensor todirect the image sensor to provide the image information of only linesof pixels needed for the one or more sub-images.

The cropping data may contain spatial resolution information. Theprocessing circuitry may be configured to determine the spatialresolution of the one or more sub-images using the spatial resolutioninformation.

The image processing circuitry may be further configured to control theimage sensor to produce image information with reduced spatialresolution by skipping analog-to-digital conversion of picture elementdetectors not needed for the one or more sub-images.

The image processing circuitry may be further configured to producedifferent sub-images with differing spatial resolutions.

The apparatus may be further comprise an image processor configured tocombine two or more sub-images representative of a common image objectand taken at different times and using different exposure times forforming one or more high-dynamic range images.

According to a second example aspect of the present invention, there isprovided a method comprising:

receiving image information from an image sensor relating to one imageframe; and

producing one or more sub-images from the received image informationaccording to given cropping data; and

forwarding the produced one or more sub-images through an output.

According to a third example aspect of the present invention, there isprovided a non-transitory memory medium comprising embodied therein acomputer program comprising:

computer code for receiving image information from an image sensorrelating to one image frame; and

computer code for producing one or more sub-images from the receivedimage information according to given cropping data; and

computer code for forwarding the produced one or more sub-images throughan output;

when executed by a computer.

According to a fourth example aspect of the present invention, there isprovided an apparatus comprising:

input means for receiving image information from an image sensorrelating to one image frame; and

processing means for producing one or more sub-images from the receivedimage information according to given cropping data and for forward theproduced one or more sub-images through an output means for outputtinginformation.

According to a fifth example aspect of the present invention, there isprovided an apparatus comprising:

an input configured to receive pixel signals from an image sensor havinga given image sensing area, the image sensor comprising pixel detectors;

an output configured to output image information for subsequent use; and

an image processing circuitry configured to:

-   -   receive the pixel signals from the input;    -   produce, based on the pixel signals, a first digital image        corresponding to a first region of the image sensing area;    -   produce, based on the pixel signals, a second digital image        corresponding to a second region of the image sensing area;    -   cause outputting with the output the first digital images with a        first resolution and first rate; and    -   cause outputting with the output the second digital images with        a second resolution and second rate; wherein at least one of the        first resolution and first rate is higher than respective second        resolution and second rate.

Term resolution may refer to spatial resolution i.e. to the accuracy inwhich objects in images are reproduced.

The image processing circuitry may comprise at least one of a digitalsignal processor, a microprocessor, an application specific integratedcircuit and an analog-to-digital converter.

The analog-to-digital converter may be configured to convert analogpixel signals into digital pixel signals only for regions belonging to aregion from which a digital image is being produced.

The first digital images may be allocated more data transfer capacitythan digital images corresponding to an image area outside the areacorresponding to the first digital images.

The second region of the image sensor area may correspond to the portionof the image sensing that is usable for producing digital images. Thesecond region may exclude the first region.

Bandwidth needed for data transfer may be further reduced by producingthe second digital images such that the image sensing area used forproducing the first digital images is excluded from the second digitalimages.

The image processing circuitry may be configured to maintain regioninformation. The region information may contain cropping data thatdefines the first region. The image processing circuitry may beconfigured to receive update instructions and to correspondingly changethe first region. The region information may further contain informationdefining the first rate. The region information may further containinformation defining the first resolution.

The region information may further comprise cropping data defining thesecond region. The image processing circuitry may be configured toreceive update instructions and to correspondingly change the secondregion. The region information may further contain information definingthe second rate. The region information may further contain informationdefining the second resolution.

The image processing circuitry may be further configured to:

produce, based on the pixel signals, a third digital image correspondingto a third region of the image sensing area; and

cause outputting with the output the third digital images with a thirdresolution and third rate;

wherein at least one of the third resolution and third rate is higherthan respective second resolution and second rate.

The region information may further comprise cropping data defining thethird region. The image processing circuitry may be configured toreceive update instructions and to correspondingly change the secondregion. The region information may further contain information definingthe third rate. The region information may further contain informationdefining the third resolution.

The third region may overlap with the first region. Alternatively, thethird region may be configured to exclude the first region at least whenthe first and third resolutions are similar.

The apparatus may be configured to produce still images.

The apparatus may be configured to produce video images.

The apparatus may be configured to concurrently produce video image ofone or more image regions and one or more still images of one or moreimage regions.

According to a sixth example aspect of the present invention, there isprovided an apparatus comprising:

an input configured to receive information from an image sensor relatingto one image frame; and

an image processing circuitry configured to produce one or moresub-images from the received information according to given croppingdata and to forward the produced one or more sub-images through anoutput.

The apparatus may be configured to send image information only with theproduced one or more sub-images.

The output may be a camera unit interface configured to exchangeinformation between a camera unit that comprises the apparatus and acircuitry external to the camera unit.

According to a seventh example aspect of the present invention, there isprovided method comprising:

receiving pixel signals from an image sensor having a given imagesensing area, the image sensor comprising pixel detectors;

producing, based on the pixel signals, a first digital imagecorresponding to a first region of the image sensing area;

producing, based on the pixel signals, a second digital imagecorresponding to a second region of the image sensing area;

outputting the first digital images with a first resolution and firstrate; and

outputting the second digital images with a second resolution and secondrate; wherein at least one of the first resolution and first rate ishigher than respective second resolution and second rate.

According to an eighth example aspect of the present invention, there isprovided a computer program comprising computer executable program code,configured to cause an apparatus, when executing the program code, toperform the method of the second or seventh example aspect.

The memory medium may be a non-transitory memory medium. The memorymedium may comprise a digital data storage such as a data disc ordiskette, optical storage, magnetic storage, holographic storage,opto-magnetic storage, phase-change memory, resistive random accessmemory, magnetic random access memory, solid-electrolyte memory,ferroelectric random access memory, organic memory or polymer memory.The memory medium may be formed into a device without other substantialfunctions than storing memory or it may be formed as part of a devicewith other functions, including but not limited to a memory of acomputer, a chip set, and a sub assembly of an electronic device.

Different non-binding example aspects and embodiments of the presentinvention have been illustrated in the foregoing. The above embodimentsare used merely to explain selected aspects or steps that may beutilized in implementations of the present invention.

Some example embodiments of the invention may be presented only thedetailed description of this document or in this summary, and/or onlywith reference to certain example aspects of the invention. It should beappreciated that embodiments relating to one example aspect or oneembodiment may apply to other example aspects or embodiments as well.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of example embodiments of the presentinvention, reference is now made to the following descriptions taken inconnection with the accompanying drawings in which:

FIG. 1 shows a schematic system for use as a reference with which someexample embodiments of the invention can be explained;

FIG. 2 shows a block diagram of an apparatus of an example embodiment ofthe invention;

FIG. 3 shows a block diagram of a camera unit of an example embodimentof the invention; and

FIG. 4 shows an example of concurrent four imaging regions.

DETAILED DESCRIPTION OF THE DRAWINGS

An example embodiment of the present invention and its potentialadvantages are understood by referring to FIGS. 1 through 4 of thedrawings.

Various examples will next be described to illustrate different exampleembodiments of the invention. The structures of these embodiments may bevaried in many ways. It is intended to make reference to a schematicsystem presented in FIG. 1 in order to present a basic system in whichexample embodiments are described, and to then discuss variousoperations according to different example embodiments. This detaileddescription merely presents some example embodiments of the invention.

FIG. 1 shows a schematic system 100 for use as a reference with whichsome example embodiments of the invention can be explained. The system100 comprises a device 110 such as a camera phone, gaming device,security camera device, personal digital assistant, tablet computer or adigital camera having a camera unit 120 with a field of view 130. Thedevice 110 further comprises a display 140. FIG. 1 also shows an imageobject 150 that is being imaged by the camera unit 120 and a background160 such as a curtain behind the image object.

In FIG. 1, the image object 150 is relatively small in comparison to thefield of view at the image object 150. Next to the image object 150,there is a continuous background 160. While this setting is not by anymeans necessary, it serves to simplify FIG. 1 and description of someexample embodiments of the invention.

FIG. 2 shows a block diagram of an apparatus 200 of an exampleembodiment of the invention. The apparatus 200 is suited for operatingas the device 110. The apparatus 200 comprises a communication interface220, a host processor 210 coupled to the communication interface module220, and a memory 240 coupled to the host processor 210.

The memory 240 comprises a work memory and a non-volatile memory such asa read-only memory, flash memory, optical or magnetic memory. In thememory 240, typically at least initially in the non-volatile memory,there is stored software 250 operable to be loaded into and executed bythe host processor 210. The software 250 may comprise one or moresoftware modules and can be in the form of a computer program productthat is software stored in a memory medium. The apparatus 200 furthercomprises a camera unit 260 and a viewfinder 270 each coupled to thehost processor 210. The camera unit 260 and the processor 210 areconnected via a camera interface 280.

Term host processor refers to a processor in the apparatus 200 indistinction of one or more processors in the camera unit 260, referredto as camera processor(s) 330 in FIG. 3. Depending on implementation,different example embodiments of the invention share processing of imageinformation and control of the camera unit 260 differently between thecamera unit and one or more processors outside the camera unit. Also,the processing is performed on the fly in one example embodiment andwith buffering in another example embodiment. It is also possible that agiven amount of images or image information can be processed on the flyand after than buffered operation mode is used as in one exampleembodiment.

It shall be understood that any coupling in this document refers tofunctional or operational coupling; there may be intervening componentsor circuitries in between coupled elements.

The communication interface module 220 is configured to provide localcommunications over one or more local links. The links may be wiredand/or wireless links. The communication interface 220 may further oralternatively implement telecommunication links suited for establishinglinks with other users or for data transfer (e.g. using the Internet).Such telecommunication links may be links using any of: wireless localarea network links, Bluetooth, ultra-wideband, cellular or satellitecommunication links. The communication interface 220 may be integratedinto the apparatus 200 or into an adapter, card or the like that may beinserted into a suitable slot or port of the apparatus 200. While FIG. 2shows one communication interface 220, the apparatus may comprise aplurality of communication interfaces 220.

The host processor 210 is, for instance, a central processing unit(CPU), a microprocessor, a digital signal processor (DSP), a graphicsprocessing unit, an application specific integrated circuit (ASIC), afield programmable gate array, a microcontroller or a combination ofsuch elements. FIG. 2 shows one host processor 210, but the apparatus200 may comprise a plurality of host processors.

As mentioned in the foregoing, the memory 240 may comprise volatile anda non-volatile memory, such as a read-only memory (ROM), a programmableread-only memory (PROM), erasable programmable read-only memory (EPROM),a random-access memory (RAM), a flash memory, a data disk, an opticalstorage, a magnetic storage, a smart card, or the like. In some exampleembodiments, only volatile or non-volatile memory is present in theapparatus 200. Moreover, in some example embodiments, the apparatuscomprises a plurality of memories. In some example embodiments, variouselements are integrated. For instance, the memory 240 can be constructedas a part of the apparatus 200 or inserted into a slot, port, or thelike. Further still, the memory 240 may serve the sole purpose ofstoring data, or it may be constructed as a part of an apparatus servingother purposes, such as processing data. Similar options are thinkablealso for various other elements.

A skilled person appreciates that in addition to the elements shown inFIG. 2, the apparatus 200 may comprise other elements, such asmicrophones, displays, as well as additional circuitry such as furtherinput/output (I/O) circuitries, memory chips, application-specificintegrated circuits (ASIC), processing circuitry for specific purposessuch as source coding/decoding circuitry, channel coding/decodingcircuitry, ciphering/deciphering circuitry, and the like. Additionally,the apparatus 200 may comprise a disposable or rechargeable battery (notshown) for powering the apparatus when external power if external powersupply is not available.

It is also useful to realize that the term apparatus is used in thisdocument with varying scope. In some of the broader claims and examples,the apparatus may refer to only a subset of the features presented inFIG. 2 or even be implemented without any one of the features of FIG. 2.In one example embodiment term apparatus refers to the processor 210, aninput of the processor 210 configured to receive information from thecamera unit and an output of the processor 210 configured to provideinformation to the viewfinder. In one example embodiment, the apparatusrefers to a device that receives image information from the image sensorvia a first input and produces sub-images to a second input of an imageprocessor, which image processor is any circuitry that makes use of theproduced sub-images. For instance, the image processor may comprise theprocessor 210 and the device in question may comprise the cameraprocessor 330 and the camera interface 280 shown in FIG. 3.

FIG. 3 shows a block diagram of a camera unit 260 of an exampleembodiment of the invention. The camera unit 260 comprises an objective310, an image sensor 320, a camera processor 330, a memory 340comprising data 344 and software 342 with which the camera processor 330can manage operations of the camera unit 260. The camera processor 330operates as an image processing circuitry of an example embodiment. Aninput/output or camera interface 280 is also provided to enable exchangeof information between the camera unit 260 and the host processor 210.The image sensor 320 is, for instance, a CCD or CMOS unit. In case of aCMOS unit, the image sensor 320 can also contain built-inanalog-to-digital implemented on common silicon chip with the imagesensor 320. In an alternative example embodiment, a separate A/Dconversion is provided between the image sensor 320 and the cameraprocessor 330.

The camera processor 330 takes care in particular example embodiments ofone or more of the following functions:

Pixel color interpolation;

White balance correction;

Edge enhancement;

Aspect ratio control by selecting pixel detectors to be used inproducing sub-images;

Aspect ratio control by cropping or stretching taken sub-images;

Anti-aliasing of images;

Vignetting correction;

Combining of subsequent images for high dynamic range imaging; and

Bayer reconstruction filtering;

Chromatic aberration correction;

Dust effect compensation;

Downscaling of second image region or foreground image e.g. forviewfinder use;

Defining individual pixel detectors for image capture for focusedreading of only those pixels that are needed for desired first regions;and

Defining individual lines of pixels for image capture for focusedreading of only lines of pixels needed for desired first regions.

The data 344 comprises, for example, cropping data that defines cornersof each first image region or sub-image that is desired to be captured.The cropping data is received, according to an example embodiment,through the interface 280 from the host processor 210.

It is understood that there are various different implementationsaccording to different example embodiments for the apparatus 200 as wellas for the camera module 260. For instance, the apparatus 200 can beprovided with an image accelerator configured to process informationreceived from the interface 280 that otherwise would be performed by thehost processor 210 or by the camera processor 330. Moreover, either orboth of the host processor 210 and of the camera processor 330 can beimplemented using one or more processors.

FIG. 4 shows an example of concurrent four first imaging regions, i.e.first to fourth primary imaging regions 410 to 440, respectively, forimage capture by the image sensor. Three of the first imaging regionsare aligned along common lines of pixels or image segments. The secondand third imaging regions 420, 430 have an overlapping region 425. Asshown by numbers on the right-hand side of each line of pixels, thereare lines with 0 to 3 different regions. FIG. 4 also shows a secondimaging region 450 that is the background or remainder of the image areaof the image sensor 464. While FIG. 4 shows four first imaging regions,there are example embodiments which support only one, two, three or morethan four first imaging regions.

In an example embodiment, different imaging regions may differ such thatsome imaging regions can be recorded as still images some other imagingregions can be recorded as video images or still images of differentspatial resolution. In video imaging, according to one exampleembodiment, the spatial resolution is adjusted to produce video imagesof desired size in horizontal and vertical pixels. In still imaging, thespatial resolution is also adjusted in one example embodiment to produceimages of desired image size. The adjustment of the spatial resolutioncan be performed by downscaling or upscaling imaging regions using anyof the known techniques such as skipping pixels, averaging pixels,interpolating pixels, replicating pixels and edge enhancing.

In one example embodiment, the processing of the pixel signals isdirected to only lines of pixel detectors with image regions to becaptured during one imaging period. The term imaging period refers hereto the exposure time of the imaging sensor. The imaging period may bealso shorter than normal exposure period. The images can be formed bycombining subsequent images for acquiring the desired imaging regions.

In one example embodiment, the entire image is always digitized i.e.analog-to-digital converted. In another example embodiment, the entireimage is first digitized with reduced spatial resolution: for instance,only each second or third line of pixels is converted to digital form.For producing of a second imaging region, the entire image isdown-sampled in one example embodiment. In one example embodiment, somepixels are skipped along lines of pixels on producing the entire image.In this way, a coarse image is obtained e.g. for display on a viewfinderwhen the user is not taking a photograph of any region of interest.Also, it is possible to use an image sensor with too many pixels for thespeed of analog-to-digital conversion and/or for the speed of datatransfer over the camera interface 280, when only selected pixeldetectors are actually read.

In one example embodiment, when the data transfer speed suffices, one ormore of the imaging regions are captured at the maximum data rateenabled by the exposure time and the speed of the image sensor and theresulting images are transferred over the camera interface 280. Forexample, let us assume a 50 Mpixel image sensor and the imaging regionsusing only 20% of the image sensors total pixel detectors. Assuming thatthe camera interface 280 is capable of transferring 20 Mpixels at a rateof 5 times a second, the same camera interface 280 can be used totransfer the imaging regions of interest 10 times a second. Or, all thepixels produced by the image sensor could only be transferred at a rateof 2 times per second.

In an example embodiment, an entire image is obtained from the imagesensor 320 and transferred over the camera interface 280 during a firstperiod of time with a first duration. Cropped images are transferred ata second period of time with a second period of time. The second periodof time is shorter than the first period of time. In this way, an imagecan be obtained of the entire image area of the image sensor 320relatively slowly and cropped smaller images can be obtained faster.

In an example embodiment, two or more cropped images are obtained fromthe image sensor 320 and transferred over the camera interface 280 forhigh dynamic range imaging. For instance, the host processor 210 canrequest in rapid succession short-exposure cropped images and combinetogether for high dynamic range imaging. Some of the successive croppedimages can be defined for different image area e.g. to better imagedarker areas in a combined image.

Embodiments of the present invention may be implemented in software,hardware, application logic or a combination of software, hardware andapplication logic. In an example embodiment, the application logic,software or an instruction set is maintained on any one of variousconventional computer-readable media. In the context of this document, a“computer-readable medium” may be any media or means that can contain,store, communicate, propagate or transport the instructions for use byor in connection with an instruction execution system, apparatus, ordevice, such as a computer, with one example of a computer described anddepicted in FIG. 4. A computer-readable medium may comprise acomputer-readable storage medium that may be any media or means that cancontain or store the instructions for use by or in connection with aninstruction execution system, apparatus, or device, such as a computer.

If desired, the different functions discussed herein may be performed ina different order and/or concurrently with each other. Furthermore, ifdesired, one or more of the above-described functions may be optional ormay be combined.

Although various aspects of the invention are set out in the independentclaims, other aspects of the invention comprise other combinations offeatures from the described embodiments and/or the dependent claims withthe features of the independent claims, and not solely the combinationsexplicitly set out in the claims.

It is also noted herein that while the above describes exampleembodiments of the invention, these descriptions should not be viewed ina limiting sense. Rather, there are several variations and modificationswhich may be made without departing from the scope of the presentinvention as defined in the appended claims.

1. An apparatus, comprising: an input configured to receive imageinformation from an image sensor relating to one image frame; and animage processing circuitry configured to produce one or more sub-imagesfrom the received image information according to given cropping data andto forward the produced one or more sub-images through an output.
 2. Theapparatus of claim 1, wherein the image sensor and the apparatus areparts of a camera unit and the output is a camera unit interfaceconfigured to exchange information between the camera unit and acircuitry external to the camera unit.
 3. The apparatus of claim 1,wherein the image processing circuitry is further configured to sendimage information only with the produced one or more sub-images.
 4. Theapparatus of claim 1, wherein the cropping data further comprisesexposure definition information and the image processing circuitry isfurther configured to control the image sensor to produce imageinformation with exposure time corresponding to the exposure definition.5. The apparatus of claim 1, wherein the image processing circuitry isfurther configured to define individual lines of pixels for imagecapture for the image sensor to direct the image sensor to provide theimage information of only the lines of pixels needed for the one or moresub-images.
 6. The apparatus of claim 1, wherein the cropping datacontains spatial resolution information and the processing circuitry isconfigured to determine the spatial resolution of the one or moresub-images using the spatial resolution information.
 7. The apparatus ofclaim 6, wherein the image processing circuitry is further configured tocontrol the image sensor to produce image information with reducedspatial resolution by skipping analog-to-digital conversion of pictureelement detectors not needed for the one or more sub-images.
 8. Theapparatus of claim 1, wherein the image processing circuitry is furtherconfigured to produce different sub-images with differing spatialresolutions.
 9. The apparatus of claim 1, further comprising: an imageprocessor configured to combine two or more sub-images representative ofa common image object and taken at different times and using differentexposure times for forming one or more high-dynamic range images.
 10. Amethod comprising: receiving image information from an image sensorrelating to one image frame; producing one or more sub-images from thereceived image information according to given cropping data; andforwarding the produced one or more sub-images through an output. 11.The method of claim 10, wherein image information is sent only with theproduced one or more sub-images.
 12. The method of claim 10, wherein thecropping data further comprises exposure definition information and themethod further comprises controlling the image sensor to produce imageinformation with exposure time corresponding to the exposure definition.13. The method of claim 10, further comprising defining individual linesof pixels for image capture for the image sensor to direct the imagesensor to provide the image information of only lines of pixels neededfor the one or more sub-images.
 14. The method of claim 10, wherein thecropping data contains spatial resolution information and the methodfurther comprises determining the spatial resolution of the one or moresub-images using the spatial resolution information.
 15. The apparatusof claim 14, further comprising controlling the image sensor to produceimage information with reduced spatial resolution by skippinganalog-to-digital conversion of picture element detectors not needed forthe one or more sub-images.
 16. The method of claim 10, furthercomprising producing different sub-images with differing spatialresolutions.
 17. The method of claim 10, further comprising: receivingthe one or more sub-images from the output; and combining two or moresub-images representative of a common image object and taken atdifferent times and exposure times for forming one or more high-dynamicrange images.
 18. A non-transitory memory medium comprising embodiedtherein a computer program comprising: computer code for receiving imageinformation from an image sensor relating to one image frame; computercode for producing one or more sub-images from the received imageinformation according to given cropping data; and computer code forforwarding the produced one or more sub-images through an output; whenexecuted by a computer.
 19. An apparatus comprising: input means forreceiving image information from an image sensor relating to one imageframe; and processing means for producing one or more sub-images fromthe received image information according to given cropping data and forforward the produced one or more sub-images through an output means foroutputting information.